Method for Factorizing Images of a Scene into Basis Images

ABSTRACT

A set of nonnegative lighting basis images representing a scene illuminated by a set of stationary light sources is recovered from a set of input images of the scene that were acquired by a stationary camera. Each image is illuminated by a combination of the light sources, and at least two images in the set are illuminated by different combinations. The set of input images is decomposed into the nonnegative lighting basis images and a set of indicator coefficients, wherein each lighting basis image corresponds to an appearance of the scene illuminated by one of the light sources, and wherein each indicator coefficient indicates a contribution of one of the light sources to one of the input images.

RELATED APPLICATIONS

This U.S. Patent Application is related to U.S. Patent ApplicationMERL-2630 cofiled herewith and incorporated by :reference.

FIELD OF THE INVENTION

This invention, relates generally to computer vision, and moreparticularly to factorizing images of a scene that is subject tolighting variations into basis images,

BACKGROUND OF THE INVENTION

Edge detection is a fundamental problem in computer vision. Edgedetection provides important low level features for many applications.Edges in images of a scene can result from different causes, includingdepth discontinuities, differences in surface orientation, surfacetexture, changes in material properties, and varying lighting.

Many methods model edges as changes in low-level image properties, suchas brightness, color, and texture, within an individual image. Yet, theissue of identifying image pixels that correspond to 3D geometricboundaries, which are discrete changes in surface depth or orientation,has received less attention.

Raskar, in U.S. Pat. No. 7,295,720 B2, detects depth edges by applying ashadow-based technique using a multi-flash camera. That method appliesonly to depth discontinuities, not changes in surface normal, andrequires a controlled set of lights that encircles the lens of a camera.

3D geometric boundaries accurately represent characteristics of scenesthat can provide useful cues for a variety of tasks includingsegmentation, scene categorization, 3D reconstruction, and scene layoutrecovery.

In “Deriving intrinsic images from image sequences,” ICCV 2001, Volume:2, Page(s): 68-75 vol.2, Weiss et al. describe a sequence of images of ascene that undergoes illumination changes. Each image in the sequence isfactorized into a product of a single, constant reflectance image and animage-specific illumination image.

U.S. Pat. No. 7,756,356 describes factoring a time-lapse photographicsequence of an outdoor scene into shadow, illumination, and reflectancecomponents, which can facilitate scene modeling and editingapplications. That method assumes a single point light source atinfinity (the sun), which is moving smoothly over time, and an ambientlighting component.

In “Appearance derivatives for isonormal clustering of scenes” IEEETPAMI, 31(8):1375-1385, 2009,” Koppal et al. describe image sequencesthat are acquired by waving a distant light source around a scene. Theimages are then clustered into regions with similar surface normals.That work also assumes a single distant light source whose positionvaries smoothly over time and an orthographic camera model.

SUMMARY OF THE INVENTION

Our invention considers a set of images of a static scene (which can bein an indoor or outdoor environment), acquired by a stationary cameraunder varying illumination conditions. One objective of the invention isto detect 3D geometric boundaries from the set of images.

Another objective of the invention, is to factorize these images into aset of basis images. In these applications, the positions of the lightsources are unknown, the lights are not necessarily point sources, anddistances from the lights (and camera) to the scene cannot be assumed tobe infinite, because they are not necessarily much larger (i.e., largerby one or more orders of magnitude) than a size of the scene. Thisbreaks the assumptions of existing methods for recovering 3D structurefrom 2D images under varying lighting such as photometric stereo,structured light, and isonormal clustering, and of existing methods forfactorizing the effects of illumination from a set of images such asfactored time-lapse video and intrinsic images.

The embodiments of the invention provide a method for recoveringlighting basis images from a set of images of a static scene, capturedwith a fixed camera viewpoint, under unknown and varying illuminationconditions. In some embodiments, 3D geometric boundaries are detected.

One objective of some embodiments is to identify 3D geometric boundariesin a set of 2D images of a static scene (which can he in an indoorenvironment) that is subject, to unknown and changing illumination. Asstrictly defined herein, a 3D geometric boundary as observed in imagesof a scene is a contour that separates two surfaces in the scene wherethere is a 3D depth discontinuity, or a significant change in surfaceorientation. These boundaries can be used effectively to understand the3D layout of the scene. The 3D geometric boundary is different from a 2Dedge, such as a texture edge or shadow edge.

To distinguish 3D geometric boundaries from 2D texture edges, someembodiments of the invention analyze the illumination subspace of localappearance at each image location. This is based on the realization thatfor non-specular, e.g., Lambertian, surfaces, neighboring pixels on thesame smooth 3D surface tend to have the same relative response tolighting even though the pixels may have different colors, i.e., albedosor reflection coefficients. The reason is that in a small neighborhoodthe 3D surface is locally planar, so two points on the surface thatcorrespond to adjacent pixels have approximately the same normal. Thedistance between these two points is typically much smaller than thedistance to any of the light sources and the camera.

Based on this realization, the 3D geometric boundary detection methodcan distinguish pixels associated with 3D geometric boundaries. i.e.,pixels whose immediate neighborhoods in the image include adiscontinuity in surface normal or in depth, from pixels whoseneighborhoods may contain sharp texture or intensity boundaries butcorrespond to a a single surface.

The method formulates 3D geometric boundary detection as a per-pixelclassification problem by analyzing the illumination subspace of localappearance at each pixel location. Specifically, the method uses thedimension of the illumination subspace to indicate the presence of a 3Dgeometric boundary.

One objective of the invention is to determine a set of lighting basisimages from the set of images of a static scene subject to unknown andchanging illumination due to combinations of a set of stationary lightsources. A lighting basis image is the image that would be formed whenthe scene is illuminated by one of the individual light sources. Thelight sources do not need to point light sources. The basis imagesprovide a natural, succinct representation of the scene, withqualitative and quantitative improvement when compared with the priorart, to enable scene editing (such as relighting) and identification andremoval of shadow edges.

In some embodiments of the invention, the method for recovery oflighting basis images uses semi-binary nonnegative matrix factorization(SBNMF). SBNMF is related to nonnegative matrix factorization (NMF),which factors a nonnegative data matrix into a product of twononnegative matrices, and for which many techniques are known.

Unlike NMF, SBNMF factors a nonnegative data matrix, into a product of anonnegative matrix and a binary matrix, where a binary matrix is amatrix in which each element is either 0 or 1. That is the methodfactors a nonnegative matrix containing the images into a nonnegativematrix of lighting basis images and a binary weight matrix thatindicates which light sources are ON or OFF for each image. Therecovered set of lighting basis images provide a compact representationof the scene under varying lighting.

In some embodiments, the basis images can be used to in conjunction withthe method for 3D geometric boundary detection to distinguish shadowedges from true 3D geometry edges.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic of a method for factorizing a set of images into aset of lighting basis images according to some embodiments of theinvention;

FIG. 2 is a block diagram of the method of FIG. 1 using semi-binarynonnegative matrix factorization;

FIG. 3 a schematic of the Method of FIG. 2;

FIG. 4 is a block diagram of pseudocode of the semi-binary nonnegativematrix factorization (SBNMF);

FIGS. 5A, 5B, and 5C are schematics of Lambertian surfaces used by someembodiments of the invention;

FIG. 6 is a block, diagram of pseudocode of a geometric boundary dectionmethod according to some embodiments of the invention; and

FIG. 7 is a block diagram of pseudocode of a geometric boundary dectionmethod in which shadow edges are removed, according to some embodimentsof the invention; and

FIG. 8 is a block diagram of a scene editing application according toone embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As shown in FIG. 1, some embodiments of the invention provide a methodfor factorizing 110 a set of images Y 101 acquired of a scene subject tovarying lighting conditions into a set of lighting basis images V 102.The scene is lit by multiple light sources, each of which can be ON orOFF for a particular image. Each basis image in the set V corresponds tothe appearance of the scene when only a single light source (e.g.,light1, light2, . . . or light5) is ON. It should be understood that theimages in Y can be acquired at arbirary times, or in a sequence by avideo camera. It should also be understood “lighting” includes otherforms of “illumination” modalities that are additive, such as infrared(IR), ultraviolet (UV), and other frequencies in the electromagneticspectrum.

As shown in FIG. 2, for some embodiments of the invention, the set ofimages 101 is factorized 210 into nonnegative lighting basis images V201 and a binary lighting indicator matrix NV 202 using semi-binarynonnegative matrix factorization (SBNMF). FIG. 3 shows the factorizationschematically.

Generative image Model

There are l light sources illuminating a scene, with each light sourcecontrolled by an independent ON/OFF switch. If there is a group of twoor more lights that are either all ON or all OFF together (such thatthere is no input image in which one of the lights in the group is ONwhile another light in the group is OFF), then the group of lights isconsidered a single light source. For example, two lights that arecontrolled by the same switch are considered a single light source.

We assign a binary variable w_(i) to indicate, the status of each lightsource i. Then, we define a basis image v_(i) ∈

⁺ ^(R) as the image formed when only the ith light, is ON, where

⁺ represents the nonnegative real numbers. Given an image y that isilluminated by any combination of the l light sources, the image can heexpressed as a superposition of individual basis images:

$\begin{matrix}{{y = {\sum\limits_{i = 1}^{l}{w_{i}v_{i}}}},{w_{i} \in {\left\{ {0,1} \right\}.}}} & (1)\end{matrix}$

Herein, we express every image as a column vector formed by stacking thevalues of all pixels in the image into a single column.

We acquire the set Y of in images lit by various binary combinations ofthe l light sources, and arrange the image data into a matrix

Y=[y ₁ , y ₂ , . . . , y _(m)] ∈

⁺ ^(n×m) .

Following equation (1), this data matrix can be factorized as:

Y=VW,   (2)

where the columns of V ∈

⁺ ^(n×l) correspond to the l basis images v_(i), and W ∈ {0,1}^(l×m) isa binary indicator matrix in which each entry W_(ij) is an indicatorcoefficient that represents the contribution of the ith light source tothe jth input image, as shown in FIGS. 2-3. That is to say, ever y inputimage is a binary combination of the in lighting, basis images.

Note that if there is ambient lighting in the scene (light that ispresent in every image), in our model this can be modeled by anadditional, basis image (an additional column in the matrix V) and acorresponding additional row of the indicator matrix W whose elementsare all 1.

In some embodiments, if a single light source is moved to a differentposition between the times at which two input images are acquired, weconsider that light source to be two separate light sources. Forexample, if two input images acquired several hours apart are bothilluminated by the sun, we consider the sun in the first image to be afirst light source and the sun in the second image to be a second lightsource.

Recovering Basis Images via SBNMF

In some embodiments, we recover the lighting basis images and indicatormatrix from the set of input images using SBNMF. If the true lightingbasis images are linearly independent, and we observe sufficientillumination variability, i.e., the rank of the true indicator matrix Wis not less than the number of lights, then the number of lights l inthe scene is given by the rank of the data matrix Y.

We formulate recovery of the basis images and indicator matrix as aconstrained optimization problem:

$\begin{matrix}{{\min\limits_{V,W}{{Y - {VW}}}_{F}^{2}},{{{such}\mspace{14mu} {that}\mspace{14mu} V_{ij}} \geq 0},{W_{jk} \in \left\{ {0,1} \right\}},{\forall i},j,k} & (3)\end{matrix}$

which we call the SBNMF. This is a challenging problem due to thenon-convex objective function and the binary constraints on W.Therefore, we initially solve a continuous relaxation:

$\begin{matrix}{{\min\limits_{V,W}{{Y - {VW}}}_{F}^{2}},{{{such}\mspace{14mu} {that}\mspace{14mu} V_{ij}} \geq 0},{0 \leq W_{jk} \leq 1},{\forall i},j,k} & (4)\end{matrix}$

where the binary constraints on W_(ij) are replaced by simple boxconstraints of lower and upper bounds This is a bi-convex problem whichwe solve using alternating direction method of multipliers (ADMM). ADMMis a variant of the conventional augmented Lagrangian method, see below.

We rewrite equation (4) using, an auxiliary variable X, and replacingpositivity and box constraints by indicator functions:

$\begin{matrix}{{{\min\limits_{X,V,W}{{Y - X}}_{F}^{2}} + {I_{\lbrack{0,\infty})}(V)} + {I_{\lbrack{0,1}\rbrack}(W)}}\mspace{14mu} {{{{such}\mspace{14mu} {that}\mspace{14mu} X} - {VW}} = 0}} & (5)\end{matrix}$

where an indicator function I_(S)(x) takes value 0 if x ∈ S and equals ∞everywhere else.

Next, we form the augmented Lagrangian:

L(X, V, W, U)∥Y−X∥ _(F) ² +I _([0,∞()(V)+I _([0,1])(W)+(μ/2)∥X−VW+U∥_(F) ²−(μ/2)∥U∥ _(F) ²   (6)

where U is the scaled dual variable and μ is the augmented Lagrangianparameter. We use the scaled form of the augmented Lagrangian functionin which the scaled Lagrangian multiplier is redefined as U=Z/μ, where Zis the original Lagrange multiplier.

ADMM solves the augmented Lagrangian dual function by a set of convexsubproblems where the biconvex function is decoupled:

$\begin{matrix}{{\left( {X^{k + 1},V^{k + 1}} \right) = {{\underset{X,{V \geq 0}}{\arg \; \min}{{X - Y}}_{F}^{2}} + {\left( {\mu \; 2} \right){{X - {VW}^{k} + U^{k}}}_{F}^{2}}}},} & (7) \\{W^{k + 1} = {\underset{0 \leq W \leq 1}{\arg \; \min}{{X^{k + 1} - {V^{k + 1}W} + U}}_{F}^{2}}} & (8) \\{U^{k + 1} = {U^{k} + X^{k + 1} - {V^{k + 1}W^{k + 1}}}} & (9)\end{matrix}$

These subproblems are iteratively solved until convergence of primal anddual residuals.

Following that, we round each entry of the matrix W to {0, 1}, anddetermine the basis images V based on the binary indicator matrix usingnonnegative least squares:

$\begin{matrix}{{{\min\limits_{V}{{{Y - {VW}}}_{F}^{2}\mspace{14mu} {such}\mspace{14mu} {that}\mspace{14mu} V_{ij}}} \geq 0},{\forall i},{j.}} & (10)\end{matrix}$

Note that because W is constant in the optimization (10), the problem isconvex.

FIG. 4 shows the pseudocode for our factorizing procedure. The variablesand equations referenced in the pseudocode in this and other proceduresshown as pseudocode are all detailed herein.

Note that in other embodiments, the individual light sources are notrestricted to being only ON or OFF, but: the light sources' intensitiescan change continuously, (e.g., a dimmer switch is used). In this case,the indicator coefficients in the indicator matrix W are not restrictedto binary values {0, 1} but can be any nonnegative real numbers. In thiscase, every input image is a nonnegative linear combination of the mlighting basis images. In such cases, the factorization can be performedusing conventional nonnegative matrix factorization.

Detecting 3D Geometric Boundaries

In some embodiments, we detect 3D geometric boundaries in the set ofimages acquired of the scene. As observed in images of the scene, 3Dgeometric boundaries are contours that separate two surfaces in thescene where there is a 3D depth discontinuity, or where there issignificant change in surface normals. For typical indoor scenes, adistant lighting assumption is not valid. To allow for nearby lighting,we consider one small image patch at a time, and analyze how the localappearance of that patch varies with multiple lighting conditions. Themethod can use patches with a variety of shapes and sizes. For example,we can consider a square or circular patch with a fixed diameter (e.g.,7 pixels) centered at each image pixel.

If all pixels in a patch come from a single smooth surface in the scene,then the patch appearance across varying lighting forms aone-dimensional subspace. If the patch contains a 3D geometric boundary,then the appearance subspace of the patch generally has dimensiongreater than one.

Illumination Subspace of Local Appearance

For simplicity, we justify our method for Lambertian surfaces with onlya direct lighting component, but an analogous argument applies to abroader class of reflectance functions and indirect lighting, e.g.,multiple reflections. To simplify the explanation, we describe onlypoint light sources, because an extended isotropic light source can bearbitrarily well approximated as a superposition of multiple point lightsources.

FIGS. 5A, 5B, and 5C schematically show three Lambertian surfaces501-503 as would be observed, by a camera 500, each illuminated by twopoint light sources A and B. For all three surfaces, the pixelscorresponding to points 1 and 2 in images acquired by the camera will benear each other. In FIG. 5A, there are no 3D geometric boundaries in thesurface 501. In FIG. 5B, there is a discontinuity in the surface normalsof the surface 502. In FIG. 5C, there is a discontinuity in depth in thesurface 503 from the camera's point of view.

We describe our notation for light source A, and that for source B isanalogous. The surface normal at point i is {circumflex over (n)}_(i),and the vector from point i to light A is r_(i) ^(a) (the correspondingunit vector is {circumflex over (r)}_(i) ^(a)). The intensity of thepoint on the image plane that corresponds to surface point i is I_(i)^(a) (for light source A) or I_(i) ^(b) (for light B):

$\begin{matrix}{{I_{i}^{a} = {\gamma_{i}^{a}\frac{{\hat{n}}_{i}^{T}{\hat{r}}_{i}^{a}}{{r_{i}^{a}}^{2}}E^{a}\rho_{i}}},{I_{i}^{b} = {\gamma_{i}^{b}\frac{{\hat{n}}_{i}^{T}{\hat{r}}_{i}^{b}}{{r_{i}^{b}}^{2}}E^{b}{\rho_{i}.}}}} & (11)\end{matrix}$

Here {circumflex over (n)}_(i) ^(T){circumflex over (r)}_(i) ^(a) is thecosine of the angle between {circumflex over (n)}_(i) and r_(i) ^(a),E^(a) is the radiance intensity of light source A, and ρ^(i) is thesurface albedo at point i. The binary value γ_(i) ^(a)=1 if point i isilluminated by source A whereas γ_(i) ^(a)=0 if point i is notilluminated by source A due to an attached or cast shadow.

For each of the three surfaces, points 1 and 2 are near each other fromthe perspective of the camera, so the points will both he included inthe same small image patch. In FIG. 5A, the patch contains no suddenchanges in normal and no depth discontinuities. Thus, the 3D distancebetween points 1 and 2 is small compared to the distances from eachpoint to each light, and hence, we have the following approximateequalities:

{circumflex over (n)}₁≈{circumflex over (n)}₂, r₁ ^(a)≈r₂ ^(a), r₁^(b)≈r₂ ^(b).   (12)

Because all points in the patch share approximately the same normal andthe same vector to each light source, we can eliminate the subscripts iin equation (11) and use {circumflex over (n)}, r^(a), and r^(b) for allpoints in the patch. For now, we assume that every point i in the patchshares a single value for γ_(i) ^(a) (which we call γ^(a)) and shares asingle value γ^(b) of γ_(i) ^(b), which means that for each lightsource, the entire patch is either illuminated by or shadowed from thatlight, i.e., the patch contains no shadow edges. We consider shadowedges below.

Let P^(a) and P^(b) represent the vector of pixel intensities of thepatch imaged under light A alone and light B alone, respectively. Forthe case in FIG. 5A, we have the approximate equality P^(a)=k^(a)ρ:

$\begin{matrix}{{\underset{P^{a}}{\begin{bmatrix}I_{1}^{a} \\I_{2}^{a} \\\vdots\end{bmatrix}} \approx {\underset{\underset{k^{a}}{}}{\frac{\gamma^{a}E^{a}{\hat{n}}^{T}{\hat{r}}^{a}}{{r^{a}}^{2}}}\underset{\rho}{\begin{bmatrix}\rho_{1} \\\rho_{2} \\\vdots\end{bmatrix}}}},} & (13)\end{matrix}$

where the scalar k^(a) is constant for all pixels in the patch, and ρ isthe vector of surface albedos for all of the pixels in the patch. Forthe same patch under light source B, we have the analogous equation:P^(b) 32 k^(b)ρ.

Thus, if a patch contains no sudden changes in normal nor in depth (andno shadow edges), then the pixel intensities under any light source areequal to a scalar multiple of ρ. In other words, the subspace of spannedby the appearance of that local patch under all light sources (which wecall the illumination subspace of local appearance) is one-dimensional(ID). Note that this is true regardless of the surface texture (albedo).Even if the surface albedo of the patch contains high-contrast textureedges, its illumination subspace of local appearance is still 1D.

This realization is at the heart of our method for finding geometricedges, because the same is not generally true if a patch contains a 3Dgeometric edge.

For example, if a patch contains an abrupt change in normal, as in FIG.5B, then the first approximation in equation (12) does not hold, and theintensity of each point in the patch depends on the cosine of the anglebetween its surface normal and its direction to the light source. If apatch contains a different type of 3D geometric boundary, a depthdiscontinuity as in FIG. 5C, then the last two approximations in (12) donot hold (because the lights are not at infinity), and the intensity ofeach point in the patch depends on the cosine of the angle between itssurface normal and its direction to the light source. In general, if apatch contains a 3D geometric edge, then its illumination subspace oflocal appearance has a dimension greater than 1.

Confidence Map of 3D Geometric Boundaries

In some embodiments, we detect geometric boundaries by identifyingpatches whose illumination subspaces of local appearance have dimensiongreater than one. For each pixel location, we extract a τ-pixel patchcentered at that location from all m input images (m lightcombinations), and arrange the patches as column vectors in a τ×mmatrix, Z:

Z=[P ⁽¹⁾ , P ⁽²⁾ , . . . , P ^((m))],   (14)

where vector P^((i)) contains all τ pixel (color or intensity) values ofthe patch extracted from image i at that pixel location. To determinethe rank of the illumination subspace of local appearance for that patchlocation, we apply singular value decomposition (SVD) to Z and obtainthe singular values {σ_(i) ^(P)} (ordered largest to smallest). In theabsence of noise, a one-dimensional illumination subspace yields justone nonzero singular value σ₁ ^(P), with σ₂ ^(P)=0. Due to noise in theimages σ₂ ^(P) is not exactly 0, but approximately 0. To determinewhether the illumination subspace of local appearance has rank 1 we usea confidence value that is is accurate in the presense of noise.

In some embodiments, for each pixel location, we determine a confidencevalue that the corresponding patch contains a 3D geometric boundary as aratio of the second to the first singular value for the patch centeredat that location:

c(P)=σ₂ ^(P)/σ₁ ^(P).   (15)

Using, equation (15), we obtain a confidence map, an image in which theintensity of each pixel is the confidence value that was determined forthat pixel location.

In other embodiments, the confidence value that the illuminationsubspace of local appearance has rank greater than 1 can be computed inother ways than equation (15). For example, we could define c(P) someother function of the singular values, such as c(P)=σ₂ ^(P)/k, where kis a normalization factor determined from the singular values of theillumination subspaces of all of the patches. The pseudocode for our 3Dgeometric boundary detection procedure in shown in FIG. 6.

In one embodiment, rather than extracting patches from the in originalinput images, we extract patches from the l nonnegative lighting basisimages described above. This can be a more stable approach if the set ofinput images is unbalanced, for example, if a large number of inputimages come from a single lighting condition, and only a few inputimages come from the other lighting conditions.

Removing Shadow Edges

Our method successfully detects both types of 3D geometric boundaries:discontinuity in the normal, and discontinuity in depth, herein bothtypes are characterized as “boundaries.” In addition, our method is notconfused by texture edges. However, shadow edges can be detected by themethod outlined in FIG. 6 as false positive 3D geometric boundaries. Apatch contains a shadow edge if for one of the light sources, somepixels of the patch are illuminated and others are in shadow.

In most cases, each shadow edge is caused by only a single light source.Based on this observation, we can use our ability to factorize a set ofimages of a scene into single-light-source lighting basis images toeliminate most of the false positives caused by shadow edges.

We can eliminate the shadows produced by light source i by subtractingbasis image v_(i) from the set of images Y:

Y ^((i)) =Y−v _(i) w ^(i),   (16)

where w^(i) is the ith row of lighting indicator matrix W, and Y^((i))denotes the scene images re-rendered with light i turned off.

Applying our boundary detection technique on Y^((i)) results in aboundary confidence map C^((i)) in which the shadow edges resulting fromthe ith light source are eliminated. The final confidence map isaggregated by taking the minimum at each pixel location among allconfidence maps of {C^((i))}_(i=1) ^(l), so that if a shadow edgedisappears when any one of the light sources is removed, that edge willnot be present in the final confidence map.

The pseudocode for our boundary detection procedure with shadow edgesremoved is shown in FIG. 7.

In one embodiment, rather than setting Y^((i)) equal to a re-renderedversion of the original images using equation (16), instead for each iwe set Y^((i)) equal to the set of all of the lighting basis imagesother than lighting basis image i. In this embodiment, for each i, thereduced set of lighting basis images Y^((i)) contains l−1 lighting basisimages. This can be a more stable approach if the set of input images isunbalanced, for example, if a large number of input images come from asingle lighting condition, and only a few input images come from theother lighting conditions.

Scene Editing

As shown in FIG. 8, one application that can use the embodiments of theinvention is scene editing. The lighting basis images 102 enable sceneediting by directly modifying individual lighting basis images, ratherthan the acquired (input) images.

The individual lighting basis images can be edited to produce editedbasis images 712 by applying an editing function 710 which can be alinear function, such as scaling, or a nonlinear function such ashistogram equalization, gamma correction, tone mapping, or brightnessand contrast adjustment. In addition it is possible to edit a region ofthe lighting basis images, such as inserting an object or modifying thetexture. An output image 720 is constructed by applying a mergingfunction 715, such as a linear combination, to the edited basis images.The editing function can be applied to all or a part of the lightingbasis images.

In another application, the shadow and highlight, regions in a lightingbasis image can be detected by finding almost black regions andsaturated regions, respectively, in the lighting basis image. Intensity(color) values of such regions can be copied from identical locations ina different lighting basis image in which the values are not black orsaturated. The values are then modified so that the image values arecontinuous across the original shadow and highlight boundaries. Thisapproach can eliminate dark shadow and highlight regions in the scenewithout saturating or darkening the entire image.

Detected shadow regions can be used to replace the texture of a surfacewith a different texture while conforming to the illuminationinformation. The brightness of the part of the new texture that is undershadow is darkened to match the shadow information.

The steps in the methods described and shown herein can be performed ina processor connected to a memory and input/output interfaces as knownin the art. It is understood that typical digital images includemillions and millions of pixels, and that: it is impossible to processintensities or color values of this enormous magnitude mentally.

Although the invention has been described by way of examples ofpreferred embodiments, it is to be understood that various otheradaptations and modifications can be made within the spirit and scope ofthe invention. Therefore, it is the object of the appended claims tocover all such variations and modifications as conic within the truespirit and scope of the invention.

We claim:
 1. A method for recovering a set of lighting basis images froma set of two or more input images of a scene illuminated by a set of twoor more light sources, wherein each light source is stationary,comprising: decomposing the set of input images into the lighting basisimages and a set of indicator coefficients, wherein each lighting basisimage is a nonnegative image that corresponds to an appearance of thescene illuminated by only one of the light sources, and wherein eachindicator coefficient indicates a contribution of one of the lightsources to one of the input images, wherein the set of input images isacquired by a stationary camera, wherein in each input image the sceneis illuminated by a combination of the light sources, and wherein atleast two images in the set of input images are illuminated by differentcombinations of the light sources, where the decomposing is performed ina processor.
 2. The method of claim 1, wherein the scene is in an indoorenvironment.
 3. The method of claim 1, wherein each indicatorcoefficient is either 0 or
 1. 4. The method of claim 1, wherein theindicator coefficients are nonnegative real numbers.
 5. The method ofclaim 1, wherein the factorizing uses semi-binary nonnegative matrixfactorization (SBNMF) to produce a matrix of nonnegative lighting basisimages and a binary matrix of indicator coefficients.
 6. The method ofclaim 5, further comprising: solving the SBNMF is using a constrainedoptimization:${\min\limits_{V,W}{{Y - {VW}}}_{F}^{2}},{{{such}\mspace{14mu} {that}\mspace{14mu} V_{ij}} \geq 0},{W_{jk} \in \left\{ {0,1} \right\}},{\forall i},j,k,$wherein min is a function returning a minimum value, Y is a matrixrepresenting the set of input images, V is the matrix of the nonnegativelighting basis images, W is the binary matrix of indicator coefficients,i indexes pixel location, j indexes the light source, and k indexes theinput images.
 7. The method of claim 6, further comprising: solving theoptimization by a continuous relaxation:${\min\limits_{V,W}{{Y - {VW}}}_{F}^{2}},{{{such}\mspace{14mu} {that}\mspace{14mu} V_{ij}} \geq 0},{0 \leq W_{jk} \leq 1},{\forall i},j,k,$where binary constraints on W_(ij) are replaced by box constraints. 8.The method of claim 7, further comprising: solving the continuousrelaxation by an alternating direction method of a multipliers.
 9. Themethod of claim 1, further comprising: editing the set of lighting basisimages to produce a set of edited basis images: and merging the set ofedited basis images to produce an output image.
 10. The method of claim9, wherein the editing uses a linear function.
 11. The method of claim9, wherein the editing uses a nonlinear function.
 12. The method ofclaim 9, wherein the merging is a linear combination of the set ofedited basis images.
 13. The method of claim 9, further comprising:detecting shadow and highlight regions in a particular lighting basisimage; and replacing intensities of pixels in each of the shadow andhighlight regions with values derived from the intensities of the pixelsin an identically located region in a different lighting basis image.14. The method of claim 9, further comprising: replacing a texture of aregion in a particular lighting basis image by a different texture inthe corresponding edited basis image; detecting a set of pixels in theregion that are in a shadow in the particular lighting basis image; anddarkening the pixels in the edited basis image that are at identicallocations to the set of pixels, in the particular lighting basis image.15. The method of claim 1, wherein distances from one or more of thelight sources to the scene are less than one or more orders of magnitudelarger than a size of the scene.
 16. The method of claim 1, wherein adistance from the camera to a part of the scene is less than one or moreorders of magnitude larger than a size of the scene.
 17. The method ofclaim 1, wherein the set of images is acquired using time-lapsephotography.
 18. The method of claim 1, wherein the camera is a videocamera, and the set of images includes images from a video acquired bythe video camera.
 19. The method of claim 1, further comprising:detecting: three-dimensional geometric boundaries in the set of inputimages.
 20. The method of claim 1, wherein the scene is in an outdoorenvironment.